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Abstract.  We  present  EMMIE  (Environment  Management  for  Multi-user  Information  En¬ 
vironments),  an  experimental  user  interface  to  a  collaborative  augmented  environment. 
Users  share  a  3D  virtual  space  and  manipulate  virtual  objects  representing  information  to 
be  discussed.  This  approach  not  only  allows  for  cooperation  in  a  shared  physical  space, 
but  also  addresses  tele-collaboration  in  physically  separate  but  virtually  shared  spaces. 
We  refer  to  EMMIE  as  a  hybrid  user  interface  because  it  combines  a  variety  of  different 
technologies  and  techniques,  including  virtual  elements  such  as  3D  widgets,  and  physical 
objects  such  as  tracked  displays  and  input  devices.  See-through  head-worn  displays  over¬ 
lay  the  virtual  environment  on  the  physical  environment.  Our  research  prototype  includes 
additional  2D  and  3D  displays,  ranging  from  palm-sized  to  wall-sized,  allowing  the  most 
appropriate  one  to  be  used  for  any  task.  Objects  can  be  moved  among  displays  (including 
across  dimensionalities)  through  drag  &  drop. 

In  analogy  to  2D  window  managers,  we  describe  a  prototype  implementation  of  a  shared 
3D  environment  manager  that  is  distributed  across  displays,  machines,  and  operating  sys¬ 
tems.  We  also  discuss  two  methods  we  are  exploring  for  handling  information  privacy  in 
such  an  environment. 
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Introduction 


In  the  early  1990s,  Weiser  coined  the  term  ubiquitous  computing  to  describe  a  world 
in  which  large  numbers  of  computing  devices  were  woven  into  the  fabric  of  our 
daily  life  [41].  These  devices  include  displays  (ranging  from  palm  sized  to  wall 
sized),  but  also  include  an  assortment  of  embedded  computers  that  add  computa¬ 
tional  behavior  to  physical  objects  and  places  that  would  not  otherwise  have  them 
(such  as  doors  or  desks).  Because  all  of  these  computers  can  be  networked  together, 
they  add  a  (mostly)  invisible  virtual  layer  to  the  physical  reality  surrounding  us  in 
our  daily  lives. 

In  contrast  to  the  proliferation  of  computing  devices  in  such  an  environment,  Aug¬ 
mented  Reality  (AR)  [4]  typically  focuses  on  the  use  of  personal  displays  (such 
as  see-through  head-worn  displays  and  headphones)  to  enhance  a  user’s  senses  by 
overlaying  a  directly  perceptible  virtual  layer  on  the  physical  world.  Because  infor¬ 
mation  is  displayed  on  a  small  number  of  displays,  computation  usually  takes  place 
on  the  few  relatively  powerful  machines  driving  those  displays.  This  contrasts  with 
the  ubiquitous  computing  paradigm,  which  is  typically  widely  distributed  and  de¬ 
centralized. 

AR  interfaces  can  enhance  a  ubiquitous  computing  environment  by  allowing  cer¬ 
tain  parts  of  the  hidden  virtual  layer  of  a  ubiquitous  computing  environment  to  be 
visualized,  as  well  as  displaying  personal  information  in  a  way  that  guarantees  it  re¬ 
mains  private  and  can  be  customized  for  each  user.  However,  one  important  draw¬ 
back  of  pure  AR  interfaces  arises  because  the  interface  elements  are  drawn  from 
purely  virtual  environments,  such  as  3D  widgets  and  3D  interaction  metaphors,  and 
thus  remain  within  the  virtual  realm.  Such  interfaces  can  be  hard  to  deal  with,  par¬ 
tially  because  the  affordances  offered  by  more  concrete  interfaces  are  absent.  As 
we  suggest  in  [27],  AR  systems  can  profit  from  the  use  of  physical  objects  and  the 
interaction  techniques  they  afford.  By  integrating  elements  of  ubiquitous  comput¬ 
ing  with  AR,  we  can  leverage  the  ubiquitous  displays  to  allow  users  to  manipulate 
information  in  a  concrete  way  when  appropriate. 

In  this  paper,  we  present  the  design  of  an  experimental  hybrid  user  interface  for 
CSCW  that  combines  AR,  conventional  2D  GUIs,  and  elements  of  ubiquitous  com¬ 
puting.  We  use  the  term  hybrid  user  interface  to  refer  to  the  synergistic  use  of  a 
combination  of  user  interface  technologies  [18].  In  the  interface  described  here, 
see-through  head-wom  displays  are  used  in  conjunction  with  other  displays  and  de¬ 
vices,  ranging  from  hand-held  to  desktop  to  wall-sized.  Our  goal  is  to  create  an 
environment  in  which  information  displayed  on  the  3D  AR  and  conventional  2D 
displays  complements  each  other,  and  can  be  easily  moved  between  the  various 
displays. 


Design  Approach 

Our  prototype  uses  AR  as  an  encompassing  multimedia  “ether”  that  envelops  all 
users,  displays,  and  devices.  This  not  only  allows  interaction  and  display  to  take 
place  in  a  common,  shared  space,  but  also  visualizes  interactions  among  the  phys¬ 
ical  devices  that  populate  the  space.  We  address  the  often  conflicting  needs  that 
collaborating  users  have  to  focus  on  each  other  and  on  the  computer-based  tasks 
they  are  performing,  by  allowing  both  to  occupy  the  same  space.  Since  users  in¬ 
creasingly  enter  meetings  carrying  their  own  laptops  or  personal  digital  assistants 
(PDAs),  and  many  tasks  benefit  from  or  require  information  that  may  reside  on 
these  personal  machines,  we  make  them  an  intrinsic  part  of  the  interaction.  Be¬ 
cause  different  tasks  and  interaction  styles  benefit  from  the  use  of  different  displays 
and  devices,  we  have  attempted  to  create  a  unified  architecture  that  supports  a  wide 
range  of  hardware.  And,  we  have  tried  to  do  this  within  a  dynamic  collaborative 
structure  in  which  users  and  their  computers  can  freely  join  and  leave  the  group. 

The  system  that  we  are  developing  is  a  working  prototype.  Like  much  research 
that  uses  experimental  devices,  our  goal  is  not  to  suggest  a  current  practical  alter¬ 
native  to  existing  mature  technologies,  but  rather  to  explore  now,  using  commercial 
hardware,  directions  that  will  become  feasible  later  when  the  needed  technologies 
reach  maturity.  Thus,  the  head-worn  displays  we  use  are  relatively  low-resolution, 
heavy,  odd-looking,  and  insufficiently  transmissive;  the  3D  trackers  suffer  from 
limited  range  and  noise-induced  jitter;  and  adding  another  computer  to  the  network 
requires  the  familiar  tedium  of  setting  parameters  and  connecting  cables.  However, 
we  remain  confident  that  these  current  impediments  will  be  overcome  by  ongoing 
research  and  development  efforts  that  address  them;  for  example,  see-through  head- 
worn  displays  that  look  much  like  conventional  eyeglasses  [37],  accurate  wide- 
range  motion  tracking  [22,  21],  and  standards  for  mobile  wireless  data  and  voice 
networking  [15].  Therefore,  our  testbed  provides  a  way  to  explore  future  user  in¬ 
teraction  paradigms  that  will  become  increasingly  relevant  as  new  hardware  breaks 
down  these  technological  barriers. 

Environment  Management 

In  analogy  to  the  window  manager  of  a  2D  GUI,  we  use  the  term  environment  man¬ 
ager  to  describe  a  component  that  organizes  and  manages  2D  and  3D  information 
in  a  heterogeneous  world  of  many  virtual  objects,  many  displays,  and  many  users. 
Traditional  window  managers  handle  a  relatively  small  number  of  windows  on  a 
single  display  (possibly  spread  across  multiple  screens)  for  a  single  user.  In  con¬ 
trast,  an  environment  manager  must  address  the  more  complex  task  of  managing  a 
global  3D  space  with  a  combination  of  virtual  and  real  objects,  and  a  heterogeneous 
set  of  computers,  displays  and  devices,  shared  by  multiple  interacting  users. 

In  this  paper,  we  introduce  EMMIE  (Environment  Management  for  Multi-user 
Information  Environments),  a  prototype  environment  manager.  EMMIE  supports  a 


Figure  1 :  A  meeting  situation  using  EMMIE.  Two  users  wear  tracked  see-through  head- worn 
displays,  one  of  whom  has  also  brought  in  his  own  laptop.  The  laptop  and  a  stylus-based 
display  propped  up  on  the  desk  are  also  tracked.  All  users  can  see  a  wall-sized  projection 
display.  The  triangular  source  for  one  tracker  is  mounted  at  the  left  of  the  table;  additional 
ceiling-mounted  trackers  are  not  visible  here. 

dynamically  changing  mix  of  displays  and  devices,  allows  information  to  be  passed 
between  2D  and  3D  devices,  and  provides  mechanisms  for  handling  privacy  in 
multi-user  environments  and  services  such  as  searching. 

A  collaboration  scenario 

We  developed  EMMIE  to  experiment  with  supporting  collaboration  among  partic¬ 
ipants  in  a  meeting.  The  participants  share  a  3D  physical  space,  for  example  by 
sitting  around  a  table,  as  shown  in  Figure  1.  This  shared  space  contains  computers 
of  different  kinds,  such  as  workstations  and  PCs  installed  in  the  meeting  room,  as 
well  as  laptops  and  PDAs  the  participants  have  brought  with  them.  These  comput¬ 
ers  provide  displays  ranging  from  wall-sized  to  palm-sized,  and  various  interaction 
devices,  such  as  keyboards,  mice,  touch  pads,  and  pens.  Each  of  the  workstations, 
PCs,  laptops,  and  PDAs  mns  its  own  unmodified  operating  system  and  2D  GUI. 

In  addition  to  the  physical  space,  participants  also  share  a  3D  virtual  space  that 
is  overlaid  on  the  physical  space  by  means  of  AR  technology,  in  our  case  tracked, 
see-through,  head-worn,  stereo  displays.  As  shown  in  Figure  22,  the  virtual  space 

2  All  images  in  this  paper  that  show  overlaid  graphics  (Figures  2,  3,  5,  and  6)  were  shot 
directly  through  an  optical,  see-through,  head-worn  display  mounted  on  a  fiberglass  dummy  head. 
The  head's  right  eye  socket  contains  a  miniature  NTSC  video  camera.  This  makes  it  possible  to 
produce  images  that  correspond  to  what  a  user  actually  sees  through  the  display.  Fuzziness  and 
artifacts  in  the  images  are  caused  by  the  low  resolution  of  the  camera  and  the  lower  resolution  of  the 
head-worn  display,  compounded  by  the  digitization  process. 


Figure  2:  A  user  manipulates  a  3D  model  with  an  optically  tracked  hand-held  pointer.  Other 
virtual  objects  shown  include  simple  iconic  3D  representations  of  applications  (e.g.,  a  “movie 
projector”  that  can  play  videos)  and  data  objects  (e.g.,  “slides”  that  represent  still  images). 

contains  graphical  objects  that  visually  appear  to  be  located  in  the  physical  space. 

The  mapping  between  the  3D  physical  and  virtual  spaces  is  achieved  by  tracking 
relevant  physical  objects,  such  as  computers,  displays,  input  devices,  and  partici¬ 
pants,  using  a  variety  of  commercial  tracking  techniques  (infrared,  ultrasonic,  and 
magnetic).  The  3D  position  and  orientation  of  these  physical  objects  is  used  to 
control  the  behavior  of  virtual  objects  in  both  the  3D  and  2D  environments. 

In  EMMIE,  most  of  the  objects  in  the  3D  virtual  space  are  3D  icons.  They  rep¬ 
resent  information,  such  as  text,  graphics,  sound  or  animation,  much  like  the  icons 
in  a  conventional  2D  GUI.  For  example,  Figure  2  includes  simple  iconic  3D  repre¬ 
sentations  of  applications  (e.g.,  a  “movie  projector”  that  can  play  videos)  and  data 
objects  (e.g.,  “slides”  that  represent  still  images).  In  a  straightforward  adaptation  of 
2D  GUIs,  dragging  data  objects  to  application  objects  allows  them  to  be  processed. 
Other  kinds  of  virtual  objects  include  3D  widgets,  such  as  menus  or  sliders  and  3D 
models  (e.g.,  the  model  of  our  lab  that  the  user  in  Figure  2  is  holding  in  his  hand). 

An  alternative  scenario  we  kept  in  mind  while  designing  our  system,  is  the  set  of 
telecubicles  described  in  [31,  10].  In  this  environment,  each  user  sits  at  a  desk  in  the 
corner  of  a  room  where  the  two  adjacent  walls  and  the  desk  surface  are  stereo  pro¬ 
jection  displays  creating  a  CAVE-like  [13]  immersive  experience.  A  set  of  remote 
telecubicles  can  be  assembled  electronically  into  a  large  virtual  room.  A  user’s  lo¬ 
cal  cubicle  contains  both  physical  and  virtual  objects,  while  the  physical  and  virtual 
objects  in  the  remote  cubicles,  as  well  as  the  remote  users,  appear  locally  only  as 
rendered  models.  Since  rendering  can  be  selectively  omitted,  physical  objects  and 
even  users  can  be  kept  hidden  from  other  participants.  One  of  our  rationales  in  the 
design  of  EMMIE  was  to  simulate  such  a  telecubicle  environment  and  to  provide  a 


UI  design  testbed  for  it.  The  techniques  presented  in  this  work  are  thus  intended  for 
both  local  and  remote  collaboration. 

Previous  and  Related  Work 

Our  research  relates  to  and  incorporates  elements  from  work  in  different  areas:  AR, 
virtual  worlds,  ubiquitous  computing,  and  CSCW. 

Our  notion  of  a  hybrid  user  interface  is  closely  related  to  Rekimoto’s  explorations 
of  multi-computer  direct  manipulation  interfaces  [32,  33].  Like  him,  we  are  inter¬ 
ested  in  user  interfaces  that  make  it  easier  to  work  in  a  heterogeneous  computing 
environment  employing  different  devices  and  displays.  We  go  beyond  the  scenario 
that  Rekimoto  describes  in  that  our  users  can  share  a  global  AR  space  with  environ¬ 
ment  management  facilities  through  their  see-through  head-worn  displays. 

i-LAND  [23]  is  an  integrated  work  environment  supporting  cooperative  work  with 
specifically  designed  roomware  components  (electronically  enhanced  walls,  tables, 
and  chairs)  that  can  share  digital  information  via  a  physical  transportation  mecha¬ 
nism  using  passive  objects  similar  to  the  mediaBlocks  Ullmer  et  al.  propose  [39]. 
EMMIE,  on  the  other  hand,  provides  information  management  facilities  in  a  global 
AR  space,  linking  different  devices  the  user  is  already  familiar  to  (their  PDAs,  lap¬ 
tops,  or  workstations)  into  the  global  space  and  to  each  other,  supplying  virtual 
intermediate  representations  for  information  exchange. 

There  is  current  research  at  Xerox  PARC  that  focuses  on  augmenting  the  physical 
world  seamlessly  and  invisibly  with  electronic  tags  to  connect  physical  objects  with 
the  computing  environment,  essentially  forming  a  “calm”  augmented  environment 
[19,  40].  As  increasing  numbers  of  physical  objects  are  linked  to  the  world  of  com¬ 
putation,  automated  management  will  become  increasingly  important.  We  believe 
that  these  systems  could  benefit  from  an  environment  management  system. 

UNC’s  “Office  of  the  Future”  [31]  provides  a  vision  of  how  today’s  low-resolution 
AR  tracking  and  display  technologies,  such  as  those  used  in  EMMIE,  could  ulti¬ 
mately  be  replaced  with  a  combined  projection  and  tracking  system  to  better  support 
a  multi-user  collaborative  environment. 

We  recently  learned  about  the  PIT  project  at  UNC  [30],  which  presents  a  two- 
person  two-screen  stereo  display  workspace  for  collaborative  study  of  a  3D  model. 
Their  system  shares  some  overall  goals  with  ours  (shared  3D  graphics  space  and 
access  to  common  devices).  In  contrast  to  EMMIE,  it  is  currently  targeted  to  a 
specific  two-person  collaboration  task  (protein  fitting);  uses  a  fixed  set  of  displays, 
each  of  which  has  a  specific  purpose;  and  does  not  support  general  information 
exchange  mechanisms  among  the  displays. 

Open  Shared  Workspace  [26]  is  based  on  the  premise  that  continuity  with  exist¬ 
ing  individual  work  environments  is  a  key  issue  in  CSCW.  Users  of  our  environment 
also  bring  in  their  own  tools,  such  as  laptop  computers,  and  can  work  with  the  desk¬ 
top  environments  with  which  they  are  familiar.  However,  we  differ  substantially 


from  this  and  other  CSCW  work  in  that  instead  of  relying  on  video  conferencing 
(or,  for  that  matter,  virtual  3D  and  multimedia  worlds  [11,  12,  36]),  we  view  a  3D 
AR  environment  as  an  embracing  shared  virtual  space,  incorporating,  instead  of 
replacing,  existing  UI  technologies.  In  fact,  with  EMMIE’ s  AR  environment  we 
are  trying  to  achieve  seamlessness  [25]  between  different  computer  platforms,  dis¬ 
play  devices  of  different  sizes  and  dimensionalities,  and  among  different  (local  or 
remote)  users. 

Researchers  at  the  University  of  Washington  [7]  and  at  the  Technische  Universitat 
Wien  [38]  have  proposed  AR  Interfaces  for  CSCW.  Both  groups  use  see-through 
head-wom  displays  to  overlay  computer  graphics  on  the  real  world.  The  University 
of  Washington  group  also  performed  experiments  to  evaluate  user  performance  in 
an  augmented  versus  a  totally  immersed  setting  and  in  an  body- stabilized  versus 
head-stabilized  setting  [6,  8].  While  this  work  shows  the  potential  value  of  AR  for 
collaboration,  we  go  beyond  the  pure  deployment  of  AR  for  visualizing  3D  objects 
or  representing  teleconferencing  avatars  to  include  environment  management. 

Since  Fitzmaurice’s  pioneering  work  on  the  Chameleon  tracked  hand-held  display 
[20],  several  researchers  have  employed  similar  displays  as  lenses  or  “see-through” 
devices  to  overlay  computer  generated  imagery  on  the  real  world  [2,  34].  We  use 
this  technique  in  the  broader  context  of  a  hybrid  user  interface  environment  man¬ 
agement  system,  recognizing  it  as  one  of  many  valuable  tools  for  collaboration. 

Finally,  EMMIE  builds  on  our  own  previous  work  combining  2D  and  3D  in¬ 
formation  displays,  in  which  we  embedded  a  physical  2D  display  in  a  virtual  3D 
information  space  [18],  overlaid  conventional  2D  windows  on  the  3D  world  using 
a  see-through  head- worn  display  [16],  and  developed  a  wearable  outdoor  hybrid 
user  interface  that  combined  a  tracked  see-through  head-wom  display  with  a  hand¬ 
held  pen-based  computer  [17].  A  number  of  other  researchers  have  worked  on 
embedding  2D  windows  in  3D  environments,  mainly  for  making  information  ac¬ 
cessible  from  within  virtual  worlds  [14,  3].  EMMIE  extends  this  research  by  using 
see-through  displays  to  integrate  multiple,  heterogeneous  displays  into  one  unified, 
collaborative  information  space. 

Interaction  with  virtual  objects 

Virtual  objects  are  manipulated  with  3D  pointing  devices  that  combine  a  tracker 
target  and  two  buttons  to  control  a  3D  arrow.  We  use  both  the  hand-held  version 
shown  in  Figure  2,  and  one  in  the  form  of  a  ring  worn  on  the  index  finger,  which  al¬ 
lows  thumb  access  to  the  two  buttons.  An  object  is  highlighted  when  the  projection 
of  the  arrow’s  tip  intersects  the  object’s  projection  in  the  viewplane  (of  the  user’s 
dominant  eye,  in  the  case  of  our  head-wom  stereo  displays).  A  user  can  pick  up  a 
highlighted  object  by  pressing  the  first  button,  causing  the  arrow  to  turn  into  a  hand. 
The  object  can  be  moved  until  the  button  is  released,  which  drops  the  object  at  the 
pointing  device’s  current  location.  This  variation  of  the  techniques  discussed  in  [9] 


allows  easy  access  to  remote  objects. 

Certain  virtual  objects  represent  applications  or  tools  embedded  in  the  virtual 
space,  such  as  image  viewers  or  sound  players.  Dropping  a  virtual  object  of  the  ap¬ 
propriate  type  onto  a  tool  opens  the  object  (e.g.,  plays  back  a  sound  file  in  the  head- 
worn  display’s  earphones  or  displays  an  image  on  the  virtual  projection  screen  of  an 
image  viewer).  Pressing  the  second  button  in  empty  space  creates  a  pie  menu  [24] 
around  the  pointer,  from  which  one  of  a  set  of  tools  can  be  selected  and  instanced. 
Pressing  the  second  button  over  a  highlighted  data  object  immediately  creates  the 
appropriate  tool  and  opens  the  object  with  it. 

Interaction  with  physical  objects 

The  physical  objects  that  EMMIE  manages  are  the  computers  present  in  the  physical 
environment  and  their  input  devices  and  tracked  displays.  There  are  two  ways  of 
looking  at  these  computers  within  the  EMMIE  framework.  On  one  hand,  they  can 
be  seen  as  self-contained  systems  with  their  own  operating  system,  user  interface 
and  software.  For  example,  a  conventional  laptop  can  be  a  perfectly  adequate  tool 
for  displaying  and  manipulating  text  and  it  can  be  used  this  way  within  EMMIE. 

On  the  other  hand,  we  can  look  at  the  same  computers  as  the  sum  of  the  interaction 
devices  and  displays  they  provide:  keyboard,  mouse,  pen,  screen,  and  speakers. 
For  example,  in  addition  to  their  normal  use  for  displaying  data,  tracked  displays 
facilitate  an  additional  kind  of  interaction,  since  their  position  and  orientation  can 
influence  what  they  display.  This  additional  mode  can  be  used  for  some  of  the 
hybrid  interaction  techniques  we  have  developed. 

Hybrid  interaction 

By  hybrid  interaction ,  we  mean  those  forms  of  interaction  that  cut  across  different 
devices,  modalities,  and  dimensionalities  [18,  35,  32,  33].  For  example,  to  use  a 
workstation  in  the  physical  environment  and  the  wall-sized  display  connected  to  it 
to  display  an  object  in  the  3D  virtual  space,  we  have  to  provide  a  way  to  move  data 
back  and  forth  between  the  2D  desktop  of  the  workstation  and  the  3D  virtual  space 
surrounding  us. 


Figure  3:  Drag  &  drop  of  virtual  objects.  A  virtual  object  is  picked  up  using  a  3D  pointing 
device  (left  image),  dragged  to  a  laptop,  whose  spherical  bounding  volume  highlights,  and 
dropped  onto  it  (center  image).  The  object  then  appears  on  the  laptop’s  screen  (right  image). 


In  EMMIE,  this  transition  between  spaces  is  done  by  simple  drag  &  drop  mecha¬ 
nisms.  The  desktop  of  each  workstation  known  to  EMMIE  provides  a  special  icon 
representing  the  virtual  space.  By  dragging  any  regular  file  onto  this  icon,  a  corre¬ 
sponding  virtual  object  (3D  icon)  is  created  in  the  virtual  space  above  the  worksta¬ 
tion’s  display.  This  3D  virtual  object  can  now  be  manipulated  with  EMMIE’ s  tools. 
It  can  be  shared  with  another  user  by  handing  it  over,  or  it  can  be  dropped  onto 
any  workstation  managed  by  EMMIE  (see  Figure  3),  which  makes  the  correspond¬ 
ing  data  available  on  the  desktop  of  that  workstation  and  starts  up  the  application 
associated  with  its  data  type.  The  effect  of  these  mechanisms  is  similar  to  the  pick- 
and-drop  technique  presented  in  [32,  33]  with  an  important  difference:  there  is  a 
visible  and  useful  representation  for  the  data  in  the  virtual  environment  while  it  is 
being  moved  between  the  physical  machines.  (The  presence  of  this  representation 
also  raises  a  variety  of  privacy  issues,  which  we  discuss  later.) 


Figure  4:  The  same  display  tablet  can  serve  as  physical  magic  lens  and  magic  mirror.  In  the 
left  picture  a  user  not  wearing  a  HMD  is  looking  through  the  magic  lens  at  a  3D  CAD  object. 

In  the  right  picture  a  user  wearing  a  HMD  has  just  dragged  a  virtual  image  slide  in  front  of 
the  mirror  for  further  inspection. 

Another  form  of  hybrid  interaction  is  the  use  of  a  tracked  display  (in  3D  physical 
space)  for  displaying  virtual  objects  (in  the  overlaid  3D  virtual  space).  Borrowing 
the  terminology  of  [5],  we  have  used  a  tracked  flat-panel  display  to  implement  both 
a  physical  magic  lens  (inspired  by  [20])  and  a  physical  magic  mirror,  which  show 
the  3D  virtual  objects  that  can  be  seen  through  or  reflected  in  the  display,  respec¬ 
tively  (see  Figure  4).  The  physical  magic  lens  and  magic  mirror  open  a  portal  from 
the  real  world  to  the  virtual  world  for  those  EMMIE  users  who  are  not  wearing 
head-wom  displays  and  who  otherwise  cannot  see  the  3D  virtual  objects.  The  lens 
and  mirror  also  provide  additional  interaction  techniques  for  all  users;  for  example, 
allowing  otherwise  invisible  properties  of  an  object,  such  as  its  privacy  status,  to  be 
inspected  and  modified,  as  described  in  the  following  section. 

Note  that  the  tracked  flat-panel  display  is  embedded  within  the  visual  field  of  a 
user  wearing  a  see-through  head-wom  display.  Based  on  the  flat  panel’s  size  and 
pixel  count,  and  its  position  relative  to  the  user  and  virtual  objects  seen  using  it, 


the  flat  panel  can  provide  a  selective  high-resolution  view  of  any  portion  of  the  3D 
virtual  space. 

To  experiment  with  hybrid  interaction  interaction  techniques,  we  implemented  a 
simple  3D  search  function  for  virtual  objects.  A  tracked  display  (acting  simultane¬ 
ously  as  a  magic  mirror  or  lens)  presents  the  user  with  a  set  of  sliders  and  buttons 
through  which  a  subset  of  the  objects  in  the  environment  can  be  specified  by  cri¬ 
teria  such  as  their  data  type  or  size.  A  bundle  of  3D  leader  lines  in  the  virtual 
space  connects  the  tracked  display  to  the  objects  that  meet  the  specified  criteria,  as 
shown  in  Figure  5).  Since  the  leader  lines  are  virtual  objects,  they  are  visible  in 
the  see-through  head-worn  displays  as  well  as  in  the  magic  mirror.  Readjusting  the 
search  criteria  causes  the  set  of  leader  lines  to  change  interactively,  implementing  a 
dynamic  query  facility  [1]  embedded  in  the  3D  world. 


Figure  5:  A  simple  interactive  search  mechanism  creates  3D  leader  lines  emanating  from  the 
tracked  display  to  objects  satisfying  the  query. 


Privacy  management 

Privacy  is  an  important  issue  whenever  data  is  represented  in  a  shared  environment. 
Since  an  EMMIE  user  may  want  some  data  to  be  private  and  other  data  public,  we 
need  to  provide  a  way  to  view  and  modify  the  privacy  of  the  3D  virtual  objects. 
Furthermore,  because  the  best  security  measure  is  eternal  vigilance,  we  want  the 
privacy  information  to  be  either  constantly  visible  or  highly  accessible.  The  chal¬ 
lenge  in  an  AR  environment  is  to  achieve  this  without  also  being  visually  annoying 
or  outright  obstructive  of  other  interactions.  In  [10]  we  considered  the  conceptual 
design  of  two  user  interfaces  for  controlling  privacy  in  a  collaborative  teleimmer- 
sive  environment:  privacy  lamps  and  vampire  mirrors.  Here  we  present  prototype 


Figure  6:  A  user  changes  the  privacy  sta-  Figure  7:  A  user  changes  the  privacy  status 

tus  of  an  object  by  moving  a  privacy  lamp  of  an  object  on  the  vampire  mirror 

above  it 

implementations  of  both  within  EMMIE  and  discuss  how  they  satisfy  the  above 
criteria. 

Privacy  lamps  (see  Figure  6)  are  cone-shaped  virtual  light  sources  that  emit  pri¬ 
vacy  as  colored  light,  typically  red,  that  distinguishes  it  from  the  ambient  lighting 
conditions.  Any  objects  in  the  environment  that  lie  substantially  within  the  light 
cone  of  a  privacy  lamp  will  be  marked  private.  These  objects  will  also  be  rendered 
as  if  lit  by  the  lamp,  providing  direct  visual  feedback  to  the  user  about  their  privacy 
state.  Typically  these  lamps  float,  facing  downward  onto  the  world.  The  higher  the 
lamp,  the  larger  the  area  of  the  light  cone  that  intersects  with  any  plane  below  it, 
and  hence  the  more  objects  that  can  be  made  private  with  one  interaction. 

Privacy  lamps  satisfy  our  design  criteria  nicely.  Both  the  lamps  and  their  lighting 
effects  are  always  visible,  so  users  can  tell  privacy  state  at  a  glance.  The  lamps 
themselves  do  not  obscure  other  interactions,  because  they  float  above  the  normal 
workspace.  Changing  the  lighting  attributes  of  objects  adds  no  clutter  to  the  scene, 
and,  because  it  mimics  a  common  physical  phenomenon,  is  easy  to  interpret  visu¬ 
ally.  Finally,  the  lamps  make  it  easy  to  find  all  private  objects  simply  by  following 
their  beams. 

Vampire  mirrors  (see  Figure  7)  act  as  magic  mirrors  in  the  virtual  environment, 
reflecting  a  user’s  virtual  objects,  except  that  they  reflect  only  public  objects  fully. 
Private  objects  are  either  invisible  or  optionally  displayed  as  a  ghost  image.  By 
placing  a  vampire  mirror  at  the  back  of  the  workspace,  a  user  can  review  the  pri¬ 
vacy  state  of  all  objects  quickly:  only  public  objects  will  appear  bright  and  full  in 
the  mirror.  To  change  an  object’s  privacy  state,  the  user  touches  its  image  on  the 
vampire  mirror. 

As  with  the  privacy  lamps,  the  vampire  mirrors  give  us  a  means  of  viewing  and 
modifying  the  privacy  state  without  cluttering  the  scene.  Interpreting  the  mirror 
is  easy,  if  one  considers  that  it  shows  the  owner  what  other  users  can  see,  making 
it  immediately  obvious  whether  an  object  is  public  or  not.  Because  the  mirror  is 


placed  behind  objects,  it  does  not  obscure  or  impede  any  other  interaction  with 
those  objects. 

In  our  original  conception  [10],  the  vampire  mirror  was  a  virtual  object.  In  EM¬ 
MIE,  we  use  a  tracked,  touch-sensitive  LCD  panel  as  a  vampire  mirror.  Users  can 
place  it  on  the  desktop  in  a  convenient  spot  to  get  a  high-resolution  view  of  the  pri¬ 
vacy  state,  and  can  touch  the  actual  display  to  change  the  privacy  state,  giving  them 
passive  haptic  feedback. 

In  addition,  this  LCD  panel  can  be  used  as  a  privacy  lens,  allowing  the  user  to 
look  through  the  display  at  objects  to  determine  their  privacy  state.  As  with  the 
vampire  mirror,  private  objects  are  invisible  or  appear  as  ghost  images,  and  public 
objects  are  rendered  normally.  Again,  the  owner  can  toggle  an  object’s  privacy  state 
by  touching  its  image  on  the  lens.  An  advantage  of  the  lens  is  that  it  allows  users 
to  work  with  a  physical  object  close  to  their  bodies,  making  the  interaction  real  and 
comfortable. 

Implementation 

EMMIE  is  implemented  on  top  of  Coterie,  our  testbed  for  exploratory  research  in 
augmented  environments  [28].  Coterie  is  implemented  in  Modula-3,  and  runs  on 
a  variety  of  platforms  (including  many  versions  of  UNIX,  and  Windows  NT/95). 
Coterie  provides  the  programmer  with  an  environment  in  which  it  is  easy  to  rapidly 
prototype  and  experiment  with  distributed  interactive  applications.  To  that  end,  it 
supports  an  object-based  distributed  shared  memory  programming  model,  allow¬ 
ing  the  programmer  to  implement  distributed  applications  much  as  they  would  im¬ 
plement  multi-threaded  applications  in  a  single  process.  Communication  is  done 
through  shared  objects,  which  may  exist  at  one  site  and  be  accessed  remotely  from 
all  others,  or  be  replicated  across  multiple  sites.  Replication  is  required  to  support 
the  highly  interactive  applications  we  develop,  as  data  that  is  needed  to  refresh  a 
display  many  times  per  second  must  be  local  to  the  process  refreshing  the  display. 

Coterie  presents  this  model  to  the  programmer  via  both  the  compiled  (Modula- 
3)  and  interpreted  (Repo)  languages,  as  well  as  various  libraries  for  such  things  as 
3D  graphics  and  tracker  control.  By  allowing  programmers  to  prototype  distributed 
programs  in  an  interpreted  language,  Coterie  greatly  speeds  the  development  pro¬ 
cess. 

EMMIE  takes  significant  advantage  of  two  components  of  Coterie:  Repo,  the  in¬ 
terpreted  language,  and  Repo-3D,  the  distributed  3D  graphics  library  [29].  EMMIE 
is  distributed  over  several  machines.  Its  primary  structure  is  a  simple  replicated 
object  directory  implemented  in  Repo,  similar  to  the  one  described  in  [28].  Each 
EMMIE  process  has  a  copy  of  a  shared  directory,  and  when  any  process  adds  or  re¬ 
moves  an  object,  the  others  are  notified.  In  this  way,  all  the  processes  are  peers,  with 
no  centralized  master  process  required  to  coordinate  the  application.  The  directo¬ 
ries  are  replicated  in  each  process  to  ensure  fast  access  to  the  objects  when  needed 


for  real  time  graphics  generation.  The  items  in  the  object  directory  are  well-defined 
object  structures  that  contain  all  the  information  needed  to  manipulate  them  in  any 
of  the  processes. 

One  of  the  object  components  is  a  Repo-3D  scene  graph  that  defines  the  appear¬ 
ance  of  the  object.  This  object  is  constructed  of  a  hierarchy  of  Repo-3D  choice 
groups,  each  of  which  allows  the  various  processes  in  EMMIE  to  choose  between 
the  various  possible  local  appearances  (e.g.,  highlighted  or  not,  in  a  mirror  or  on  a 
head-wom  display),  as  well  as  to  control  the  global  appearance  (e.g.,  publicly  vis¬ 
ible  or  private  to  one  process).  Because  this  single  well-defined  object  hierarchy 
is  replicated  in  all  processes  that  import  the  object,  the  clients  can  be  defined  in  a 
straightforward  manner,  and  various  interaction  techniques  and  object  representa¬ 
tions  can  be  experimented  with  simply  and  cleanly. 


Figure  8:  Architecture  of  the  EMMIE  system 


Figure  8  shows  a  diagram  of  the  architecture.  Some  users  wear  Virtual  i.O  see- 
through  head-wom  displays  with  hear-through  earphones.  Each  head- worn  display 
is  connected  to  a  3D-hardware  accelerated  PC  or  workstation,  which  also  controls 
its  user’s  3D  pointing  device.  The  3D  position  of  each  head- worn  display  and  point¬ 
ing  device  is  tracked  with  an  Origin  Instruments  DynaSight  infrared  LED  tracker 
and  each  display’s  orientation  is  tracked  with  a  built-in  magnetometer  and  incli¬ 
nometer.  The  magic  mirror  and  lens  are  implemented  on  a  Wacom  PL-300  LCD 


panel  with  pen-input  facilities,  driven  by  a  PC,  and  tracked  by  a  Logitech  6DOF 
ultrasonic  tracker.  Other  workstations  and  laptops  join  the  environment  by  running 
a  background  thread  implementing  EMMIE’s  drag  &  drop  functionality,  allowing 
them  to  be  fully  integrated  in  the  environment.  While  we  assume  the  workstation 
displays  stay  in  fixed  positions,  the  laptop  displays  are  tracked  with  Logitech  6DOF 
trackers.  Hand-held  devices  such  as  the  3Com  Palm  Pilot  are  included  by  mnning 
a  web  browser  on  them  and  sending  them  HTML  over  a  PPP  link  from  a  Coterie 
process  on  another  machine.  All  processes  of  the  distributed  system  share  access  to 
the  same  database  of  virtual  objects,  discussed  above. 

Future  Work:  Automated  3D  Layout  Assistance 

The  next  avenue  we  would  like  to  explore  is  the  use  of  dynamic,  context-sensitive 
techniques  that  can  be  added  to  an  environment  manager  to  help  users  manage  infor¬ 
mation  more  effectively.  There  are  additional  analogs  of  2D  window  management 
techniques  that  can  be  adapted  to  this  environment.  A  range  of  new  techniques  could 
also  help  the  user  deal  with  the  dynamic,  3D  nature  of  augmented  environments. 

First,  let  us  consider  the  3D  analogs  of  some  techniques  used  in  2D  window  man¬ 
agers  to  help  users  both  position  objects  when  they  are  created,  and  keep  them 
organized.  For  example,  when  a  new  item  is  created,  the  environment  manager 
should  place  it  in  a  reasonable  initial  location,  such  as  an  unoccupied  location  near 
the  focus  of  the  user’s  attention.  Some  window  managers,  such  as  Xll’s  twm,  pro¬ 
vide  such  assistance  by  positioning  new  windows  on  unoccupied  parts  of  the  screen 
(when  possible).  However,  the  definition  of  “unoccupied”  is  more  complicated  in 
an  augmented  environment  than  in  a  desktop  interface,  as  not  only  virtual  but  real 
items  must  be  considered.  We  do  not  want  to  place  a  new  virtual  item  on  top  of  some 
important  physical  item  such  as  a  telephone,  for  example,  unless  it  is  meaningfully 
associated  with  it. 

Other  techniques  could  help  the  user  deal  with  the  dynamic  nature  of  augmented 
environments.  For  example,  returning  to  the  analogy  between  desktop  window  man¬ 
agers  and  environment  managers  such  as  EMMIE,  we  note  that  window  managers 
do  exactly  what  the  user  tells  them;  no  more,  no  less.  Such  an  approach  is  fine 
for  the  desktop  interface,  as  the  desktop  itself  is  generally  static  and  lifeless,  with 
activity  happening  only  inside  statically  positioned  windows.  By  contrast,  an  envi¬ 
ronment  manager  must  be  suited  to  the  dynamic  nature  of  an  augmented  environ¬ 
ment,  interacting  with  the  user’s  actions  and  assisting  the  user  in  managing  their 
ever-changing  environment. 

On  a  traditional  2D  desktop  UI,  the  organization  of  information  typically  changes 
in  response  to  some  user  action,  such  as  starting  or  stopping  programs,  moving 
windows,  and  so  forth.  In  an  augmented  environment,  information  is  positioned  in 
3D,  often  in  relation  to  objects  that  may  move.  As  the  user,  other  people,  and  objects 
in  the  environment  move  about,  the  relationships  between  the  virtual  and  real  world 


change.  For  example,  if  a  group  of  objects  is  suspended  in  space  above  a  table,  and 
two  users  of  the  system  wish  to  talk  to  one  another,  they  most  likely  do  not  want  the 
objects  blocking  their  view  of  each  other.  Rather  than  forcing  the  users  to  move  to 
accommodate  the  virtual  objects,  we  would  instead  like  the  environment  manager  to 
move  the  virtual  objects  to  accommodate  the  users.  To  accomplish  this,  the  system 
could  make  use  of  knowledge  about  which  objects  are  currently  important  to  the 
users,  and  which  are  not.  For  example,  the  system  could  infer  that  it  is  important 
for  a  user  to  be  able  to  see  all  other  users  and  relevant  displays.  Therefore,  the 
system  could  constrain  the  virtual  objects  so  they  do  not  interfere  with  the  users’ 
views  of  each  other  and  of  their  relevant  displays. 

Conclusions 

We  have  presented  EMMIE,  a  prototype  hybrid  user  interface  to  an  augmented  in¬ 
formation  environment.  EMMIE  supports  collaborating  users  by  providing  and  co¬ 
ordinating  virtual,  physical  and  hybrid  interaction  techniques  for  different  parts  of 
the  environment.  Many  of  the  interaction  and  user  interface  techniques  presented 
are  variations  of  ones  that  have  been  proposed  before.  Our  major  contribution  is 
their  coordination  within  a  single  cohesive  framework  for  collaborative  use.  By 
merging  virtual,  physical,  and  hybrid  interaction  techniques,  each  in  the  situation 
where  it  is  most  appropriate,  a  hybrid  user  interface  is  created  whose  potential  is 
much  greater  than  that  of  the  sum  of  its  parts. 
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